Wearable Sensor-Based Real-Time Sonification of Motion and Foot Pressure in Dance Teaching and Training

July 2012 Journal of the Audio Engineering Society 60(7):580-589

In this article, we present a wearable sensor setup for motion and pressure capturing and real-time sonification of these parameters. The sensor setup is modular and adapted to the specific needs of dancers and dance teachers. Closed-loop interactive sonification in these scenarios conveys information by using nonspeech sound in real time. It addresses the sense of hearing and supports the perception of the motion, similar to the everyday experience of hearing footsteps on the ground, but even the perception of the weight distribution of the dancer on the foot. Sonification addresses the high resolution of the human auditory system and thereby enables listeners to recognize small and even invisible changes in body movement, as they manifest in distinguishable changes in level, frequency, or rhythm. As dancers are used to coordinate their movements to music, sound, and rhythm sonification in this case displays multiple parameters of their dance moves. Accordingly, different dance steps and moves lead to typical acoustic "fingerprints" that accompany the movements in real time. The presented auditory display supports the understanding, coordination, and learning of dance steps, gestures, and movements. In our setup, the data is captured from a wearable sensor kit and is sonified in real time with SuperCollider. Specifically, we use insole sensors to measure the pressure of the dancers' feet on the floor and based on this the weight distribution and the applied forces in jumps or steps. Further sensors are goniometers to detect the angle of the knee, accelerometers, and gyroscopes fixed on the foot. In the evaluation section, we examine the jumps "Sauté" and "Italien Changement" in a jump sequence from classical ballet dance repertoire. For the evaluation, the sensor setup in combination with real-time sonification is used in dance classes of different age and level.